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United States Government Accountability Office 
Washington, DC 20548 

January 9, 2009 

The Honorable Daniel E. Lungren 
Ranking Republican Member 
Subcommittee on Transportation 
and Infrastructure Protection 
Committee on Homeland Security 
House of Representatives 

The Honorable John L. Mica 

Ranking Republican Member 

Committee on Transportation and Infrastructure 

House of Representatives 

Subject: Aviation Security: TSA's Cost and Performance Study of Private-Sector Airport 
Screening 

This report formally transmits the enclosed briefing in response to your request that we 
review the Transportation Security Administration's (TSA) Screening Partnership Program 
(SPP). In accordance with the Aviation and Transportation Security Act, TSA created the SPP 
to allow commercial airports an opportunity to apply to TSA to use private sector screeners 
through qualified private-screening contractors approved by TSA. In February 2008, TSA 
issued a report on its study comparing the cost and performance of screening services at SPP 
and non-SPP airports. Our briefing addresses the following questions: (1) To what extent did 
the design of TSA's study of the cost and performance of passenger and checked baggage 
screening services at selected SPP and non-SPP airports affect the usefulness of the study? 

(2) To what extent has TSA taken actions to identify and eliminate any unnecessary 
overhead/supervisory redundancies at SPP airports between TSA and contractor personnel? 

(3) What factors do airport operators cite as having contributed to airports' decisions about 
whether to participate in the SPP? We are recommending that if TSA plans to rely on its 
comparison of costs and performance of SPP and non-SPP airports for future decision 
making, the agency update its study to address the limitations we identified, for example, by 
including various cost elements that were excluded and conducting statistical tests to 
determine the level of confidence in any observed differences in screening performance. TSA 
generally agreed with our findings and recommendation. 



As agreed with your offices, unless you publicly announce its contents earlier, we plan no 
further distribution of this report until 30 days from the date of this report. At that time, we 
will send copies of this report to the Secretary of Homeland Security and other interested 
parties. This report will also be available at no charge on our Web site at http://www.gao.gov. 
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Should you or your staff have any questions concerning this report, please contact me at 
(202) 512-3404 or berrickc@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this report. Key contributors to 
this report were Glenn Davis, Assistant Director; Chuck Bausell, Jr.; Katherine Davis; Stanley 
Kostyla; Thomas Lombardi; Frederick Lyles, Jr.; Deborah Ortega; and John Warner. 




Cathleen A. Berrick 

Managing Director, Homeland Security and Justice Issues 
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Introduction 



• Passenger screening is a critical component of the security of our nation's commercial aviation 
system. Passenger screeners, 1 both federal and private, use metal detectors, X-ray and other 
detection machines and systems, and physical searches to examine passengers and their 
baggage to identify threat objects. 



• On November 19, 2002, in accordance with the Aviation and Transportation Security Act (ATS A), 
the Transportation Security Administration (TSA) began a 2-year pilot program at five airports 
using private screening companies to screen passengers and checked baggage. 2 ATSA further 
mandates that TSA allow airports to request that the screening of airline passengers and 
baggage be conducted by private screeners in lieu of federal screeners.* 



• In November 2004 and in accordance with ATSA, TSA created the Screening Partnership 
Program (SPP) to allow all commercial airports an opportunity to apply to TSA to use private- 
sector screeners through private screening contractors approved by TSA. TSA remains 
responsible for funding all passenger and baggage screening activities at SPP airports as well as 
airports with federal screeners. 



'Federal screeners are employed by the Transportation Security Administration (TSA) and are referred to as transportation security officers; private screeners are employed by 
qualified private screening companies under contract to TSA. ^ 

2See Pub. L. No. 107-71, § 108, 115 Stat. 597, 611-13 (2001) (codified at 49 U.S.C. § 44919). 

ub. L. No. 107-71, § 108, 1 1 5 Stat, at 61 1-13 (codified as amended at 49 U.S.C. § 44920). 
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Introduction (cont'd) 



In May 2007, TSA awarded a contract to Catapult Consultants, in the amount of $442,521 , to 
conduct a cost and [performance analysis of airports with private screeners versus airports with 
federal screeners. This analysis would be used to assist senior TSA leadership with strategic 
decisions reqarding the degree to which TSA should leverage public/private .partnerships in the 
area of scree 
2007. 1 



warding the degree to which TSA should leverage public/private partnerships in the 
3ning services. The contractor issued a report on its analysis to TSA in December 



• In February 2008, TSA issued a report on a study TSA conducted comparing the cost and 
performance of screening at SPP and non-SPP airports. 2 The agency initiated this work in 
anticipation that more airports would apply to participate in the SPP, and, because of this, it 
wanted to obtain information regarding how such participation would impact the agency's budget. 
In addition. TSA also wanted to obtain information about how the existing SPP airports compared 
to non-SPP airports. 

• TSA officials stated that although they used the study prepared by Catapult Consultants to 
conduct their comparison of screening at SPP and non-SPP aijports, they consider the TSA 
study to be the official agency study. In conducting the study, TSA compared the cost of 
operating screening at SPP airports with the cost that would be incurred in the agency's budget if 
these airports were run as fullvfederal or non-SPP airports. TSA used 2007 invoice data 
submitted by the following six "SPP airports and determined what the cost would be if these 
airports were instead staffed with federal transportation security officers (TSO) (see app. I for 
additional details about TSA's approach and methodology). 

1. Greater Rochester International Airport, Rochester, New York 

2. Jackson Hole Airport, Jackson, Wyoming 

3. Joe Foss Field, Sioux Falls, South Dakota 

4. Kansas City International Airport, Kansas City, Missouri 

5. San Francisco International Airport, San Francisco, California 

6. Tupelo Regional Airport, Tupelo, Mississippi 

'Catapult Consultants, Private Screening Operations: Business Case Analysis, Transportation Security Administration, Screening Partnership Program, December 14, 2007. 
transportation Security Administration, A Report on SPP Airport Cost and Performance Analysis and Comparison to Business Case Analysis Finding, February 1 , 2008. 
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Introduction (cont'd) 



• TSA compared the performance measures of SPP airports to non-SPP airports in the 
same airport category (category X, 1,11, III, and IV). 1 Specifically, TSA analyzed 2007 
performance data for five performance measures: threat image projection detection 
rates, 2 recertification, 3 passenger waiting, peak wait times, and checkpoint capacity 
utilization. An airport was considered an average performer if the results for these 
measures fell within one standard deviation from the airport category average (the 
middle 68.2 percent of the category). 

• TSA found that screening at SPP airports currently costs approximately 17.4 percent 
more to operate than at airports with federal screeners, and that SPP airports fell 
within the "average performer" category for the performance measures included in its 
analysis. Although the TSA contractor used a different methodological approach for 
comparing the cost and performance of SPP and non-SPP airports, the contractor 
concluded that passenger screening at the SPP airports has historically cost from 9 to 
17 percent more than at non-SPP airports, and private screeners performed at a level 
that was equal to or greater than that of federal TSOs. (Additional information about 
the methodology and findings of the TSA contractor's study is contained in app. II and 
app. Ill, respectively.) 



1 TSA classities the commercial airports in the United States into one of five security risk categories (X, I, II, III, and IV) based on various factors, such as the total number ot 
takeoffs and landings annually, and other special security considerations. In general, category X airports have the largest number of passenger boardings, and category IV 
airports have the smallest. 

2 The Threat Image Projection system is designed to test TSOs' and private screeners' detection capabilities by projecting threat images, including images of guns and explosives, 
; as they are screened. TSOs and private screeners are responsible for positively identifying the threat image and calling for the bag to be searched. 
id private screeners must pass an annual standardized recertification test composed of image, job knowledge, and Standard Operating Procedures tests. 
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Introduction (cont'd) 



In presenting its findings, TSA also stated in its report that the current SPP model is 
constrained by the law 1 and TSA policy and allows for little variance in the inputs for 
staffing, compensation, and operations at SPP airports. Therefore, the outputs of such a 
constrained model produce similar performance results with an anticipated costs premium 
due to an additional layer of contractor management. 

The Ranking Republican Member of the House Committee on Transportation and 
Infrastructure and the Ranking Republican Member of the Subcommittee on Transportation 
and Infrastructure Protection, House Committee on Homeland Security, requested that we 
provide information on TSA's cost and performance study of SPP airports, any actions that 
TSA has taken to identify and prevent redundancies between TSA and contractor 
personnel, and factors that have contributed to an airport operator's decision about whether 
to participate in the SPP. 



'See, e.g., § 44920(c) (providing that private screening companies selected by TSA must provide its screening personnel compensation and other benefits at a level not less 
than the compensation and other benefits provided to federal government personnel). Q 
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Objectives 



Our objectives were to determine: 

• To what extent did the design of TSA's study of the cost and performance of passenger and 
checked-baggage screening services at selected SPP and non-SPP airports affect the 
usefulness of the study's results? 

• To what extent has TSA taken actions to identify and eliminate any unnecessary 
redundancies at SPP airports between TSA administrative staff, including supervisory 
positions, and contractor personnel? 

• What factors did airport operators cite as having contributed to their decisions about whether 
to participate in the SPP? 
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Scope and Methodology 



To determine the extent to which the design of TSA's study of the cost and performance of passenger 
and checked-baggage screening services at selected SPP and non-SPP airports affected the 
usefulness of the study's results, we 



• analyzed TSA's study and available documentation, such as calculations used in developing 
the cost estimates, to determine what costs were included in the comparisons, whether the 
comparisons accounted for possible performance differences among screeners at selected 
SPP and comparable non-SPP airports, whether the comparisons were supported by a 
workforce analysis to identify any unnecessary redundancies at SPP airports, and whether the 
study accounted explicitly for uncertainty; 



• compared TSA's study methodology as described by TSA officials to criteria in GAO's 

Standards for Internal Control in the Federal Government, 1 federal accounting standards for 
managerial cost accounting, 2 GA0 3 and Office of Management and Budget (OMB) 4 guidance for 
estimating and managing program costs, and OMB's guidance for statistical surveys, data 
quality, and performance assessments, 5 and assessed whether TSA and contractor officials 
took actions to ensure that the data used in the studies were reliable; 



1 GAO, Standards for Internal Control in the Federal Government, GAO-AIMD-00-21 .3-1 (Washington D.C.: November 1999). 
2 Federal Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards for the Federal Government. 
3 GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, GAO-07-1 134SP (Washington, D.C.: July 2007). 
OMB Circular A-76, Performance of Commercial Activities, and OMB Circular A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs. 

' "tandards and Guidelines for Statistical Surveys, September 2006; Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information 
iated by Federal Agencies, October 1, 2001; and Program Assessment Rating Tool Guidance, No. 2008-01, January 29, 2008. 
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Scope and Methodology (cont'd) 



assessed the type of performance measures that were included in the TSA and contractor studies, 
the methodology used to identify differences in private screener/TSO performance, and whether 
analyses were conducted to determine if observed differences in performance were statistically 
significant; and 

interviewed TSA and contractor officials to obtain information on the models that TSA and its 
contractor used to assess the cost of screening operations, and to assess the performance of 
private and federal screeners at the selected SPP and comparable non-SPP airports. 
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Scope and Methodology (cont'd) 



To determine the extent to which TSA has taken actions to identify and eliminate any unnecessary 
redundancies at SPP airports between TSA administrative staff, including supervisory positions, 
and contractor personnel: 

• We analyzed organizational charts, as well as information on supervisory roles and 

responsibilities, for TSA personnel at six SPP airports we visited and six non-SPP airports to 
identify possible redundancies between TSA and contractor personnel. While information we 
obtained at these locations is not generalizable across all airports, we selected these airports 
because they represent a mixture of different geographic locations, passenger and baggage 
throughput, 1 and checkpoint configurations. In addition, two of the SPP airports — Kansas 
City International Airport and San Francisco International Airport — also participated in TSA's 
2-year pilot program and, therefore, had several years of experience using private screeners. 



Table 1: Airports Included in GAO's Review 



SPP airports 


Non-SPP airports 


1. Joe Foss Field, Sioux Falls, South Dakota 


1 . Barkley Regional Airport, Paducah, Kentucky 


2. Kansas City International Airport, Kansas City, Missouri 


2. Central Illinois Regional, Bloomington, Illinois 


3. Key West International Airport, Key West, Florida 


3. Logan International Airport, Boston, Massachusetts 


4. Roswell Industrial Air Center, Ftoswell, New Mexico 


4. Missoula International Airport, Missoula, Montana 


5. San Francisco International Airport, San Francisco, California 


5. Salt Lake City International Airport, Salt Lake City, Utah 


6. Charles M. Schulz-Sonoma County Airport, Santa Rosa, California 


6. Syracuse-Hancock International Airport, Syracuse, New York 


Source: GAO 



'Throughput refers to the number of passengers and bags that are processed by TSA. 

10 
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Scope and Methodology (cont'd) 



• We assessed TSA's actions to identify unnecessary redundancies between TSA and 
contractor personnel based on GAO 1 and Office of Personnel Management (OPM) 2 guidance 
on workforce planning. 

• We interviewed TSA headquarters personnel and airport officials at the six SPP airports we 
visited to obtain their views on the extent to which redundancies were occurring between 
TSA and contractor personnel. 

• We interviewed TSA headquarters officials to discuss the agency's approach for allocating 
screening and supervisory personnel at SPP and non-SPP airports, including the extent to 
which TSA had conducted a workforce analysis. 



'GAO, Human Capital: Key Principles for Effective Strategic Workforce Planning, GAO-04-39 (Washington, D.C.: Dec. 1 1 , 2003). 
2 OPM, Workforce Planning Model, September 2005. 
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Scope and Methodology (cont'd) 



To determine what factors airport operators cite as having contributed to airports' decisions about 
whether to participate in the SPP, we 

• reviewed the results of a survey conducted by the Airports Council International — North 
America, which represents local, regional, and state governing bodies that own and operate 
commercial airports in the United States, to determine how to make the SPP more attractive to 
airports and interviewed officials from three aviation associations 1 to obtain their views on the 
SPP; 



• interviewed airport managers or airport security directors at the SPP airports we visited to 
determine what factors led to their participation in the SPP; 

• interviewed airport managers or security directors at six non-SPP airports we contacted to 
determine why they were not participating in the SPP; and 



• interviewed TSA officials to identify their efforts to promote the SPP. 



1 Air Transport Association, Airports Council International — North America, and the American Association of Airport Executives. Because we selected a nongeneralizable 
sample of aviation organizations, the information we gathered from our interviews with these organizations cannot be used to make inferences about all aviation "| 2 

lanizations. However, we believe that the information we obtained was useful in learning about how such organizations view the SPP. 
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Scope and Methodology (cont'd) 



We conducted this performance audit from August 2007 to November 2008 in accordance with 
generally accepted government auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that the evidence obtained 
provides a reasonable basis for our findings and conclusions based on our audit objectives. 



13 



Page 15 



GAO-09-27R Airport Screening 



Enclosure I: Briefing Slides 



j- G AP 

mfl^^^ Accountability * Integrity * Reliability 



Results in Brief 



TSA's study comparing the cost and performance of screening services at SPP and non-SPP airports 
had some design strengths, such as recognizing that cost savings would be limited given the 
mandated structure of the SPP and assessing whether performance data were collected in a uniform 
manner. However, based on the criteria that we used, the design was limited because TSA did not: 

• Include the impact of potential overlapping administrative staff 1 on the costs of SPP airports. TSA 
is currently conducting a workforce analysis to identify unnecessary redundancies in its staffing, 
including overlapping administrative personnel; TSA officials stated that this effort is ongoing and 
has no completion date. 

• Account for workers compensation, general liability insurance, and some retirement costs paid by 
the federal government, as well as the lost corporate income tax revenue from private screening 
contractors, when replacing private with federal screening. 

• Call for an assessment of the reliability of the costs and performance data, as called for in federal 
accounting standards. 2 

• Document the rationale for including the five performance measures reviewed in the study while 
excluding others. 

• Call for statistical analyses to determine the level of confidence in observed differences in 
performance between SPP and non-SPP airports. 



'TSA administrative staff include federal security directors, deputy federal security directors, assistant federal security directors, scheduling operations officers, stakeholder liaisons, 
training coordinators/specialists, customer service managers, industrial engineers/metric specialists, administrative officers, human resources specialists, 

financial specialists, executive secretaries, security (Office Automation), and screening managers. "| 4 




il Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards for the Federal Government. 
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Results in Brief (cont'd) 



The limitations in the design of TSA's study comparing the cost and performance of SPP and non-SPP 
airports were due to several key factors related to the study's purpose and data availability. For 
example, TSA officials stated that they did not include some cost elements in the study because they 
wanted to determine the impact of the SPP on TSA's budget, rather than to determine the impact to 
the federal government as a whole. In addition, for its comparison of performance, TSA analyzed 
measures for which information was most complete, among other things. 

Because of these limitations, we believe that TSA should not use the study as sole support for major 
policy decisions regarding the SPP. 
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Results in Brief (cont'd) 



TSA has taken actions to identify unnecessary redundancies in the duties of administrative staff 
assigned to SPP and non-SPP airports, but the agency has not completed these efforts. 

• According to identified effective practices, 1 strategic workforce planning requires, among other 
things, that agencies develop strategies that are tailored to address gaps in the number, 
deployment, and alignment of human capital approaches for enabling and sustaining the 
contributions of all critical skills and competencies. 

• TSA is conducting a workforce analysis that is intended to identify any redundancies. TSA has 
collected and analyzed information, such as data on core functions, staffing levels, hub-spoke 
responsibilities, skill sets, and collateral duties. However, steps that have not yet been 
completed include implementing revised job analysis tools to provide federal security directors 
(FSD) 2 with greater flexibility in staffing, and implementing organizational models to ensure that 
there is consistency in reporting relationships and utilization of positions. TSA officials said that 
the agency will be continuously updating its analysis. 

• TSA and SPP contractors at the airports we visited had mixed views about whether 
unnecessary redundancies at SPP airports exist. 



■GAO-04-39. 

2 FSDs are the ranking TSA authorities responsible for leading and coordinating TSA security activities at the nation's commercial airports. 
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Results in Brief (cont'd) 



Various factors have contributed to airports' decisions about whether to participate in the SPP. 

• Selected officials at the six SPP airports we visited cited various reasons for participation. 
Managers at three of the airports stated that TSA had suggested that they apply to the SPP 
program due to TSA staffing constraints, two stated that they decided to participate 
because they were part of the pilot program, and another stated that he believed that 
private screening contractors would provide better service than the federal government. 

• Officials at the six airports we contacted that have decided not to join the SPP stated that 
their decision was based largely on their satisfaction with TSA's passenger and baggage 
screening services. 
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Results in Brief (cont'd) 



We are recommending that if TSA plans to use its cost and performance study in future decision 
making regarding the SPP, it revise its study to address the methodological limitations that we have 
identified. TSA generally agreed with our findings and recommendation. 
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Background 



• ATSA was enacted on November 19, 2001 , in response to the September 1 1 , 2001 , terrorist 
attacks. ATSA established TSA and charged it with responsibility for security in all modes of 
transportation, including aviation. 

• Prior to ATSA and the establishment of TSA, passenger and baggage screening had been 
performed by private screening companies under contract to airlines. ATSA required TSA to 
create a federal workforce to assume the job of conducting passenger and checked-baggage 
screening at commercial airports. The federal workforce was to be in place by November 2002. 

• As required by ATSA, TSA began a 2-year pilot program using private screening companies to 
screen passengers and checked baggage on November 19, 2002, and selected five airports, one 
in each airport category, to participate in the program. 1 

• Although an FSD is responsible for security at every commercial airport, not every airport has an 
FSD dedicated solely to that airport. Most category X airports have an FSD responsible for that 
airport alone. Other airports are arranged in a "hub and spoke" configuration, in which an FSD is 
located at or near a hub airport but also has responsibility over one or more spoke airports of the 
same or smaller size that are generally located in geographic proximity. 



! See 49 U.S.C. §44919. 
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Background (cont'd) 



• Beginning in November 2004, in accordance with ATSA, all commercial airports with federal 
security screening became eligible to apply to the newly established SPP. 1 An airport operator 
may submit to TSA an application to have the screening of passengers and checked baggage at 
an airport carried out by screening personnel of a private screening company, under a contract 
entered into between the private screening contractor and TSA. 



• TSA acknowledged that TSA and SPP contractors are constrained by both law and TSA policy, 
which limits opportunities to make programmatic changes to improve contractor performance and 
achieve cost savings. Furthermore, although TSA staffs are smaller at airports that have SPP 
contractors, TSA continues to be responsible for overseeing screening operations at SPP 
airports and for ensuring that contract screening companies provide effective and efficient 
security operations. 



• TSA's SPP fiscal year 2009 budget is about $151.3 million. 2 



• As of September 2008, 1 commercial airports were participating in the SPP, and TSA has 
approved the applications of 7 additional airports. 

1 See 49 U.S.C. § 44920. 

2 See 152 Cong. Rec. H9792, 9797-98 (daily ed. Sept. 24, 2008) (explanatory statement accompanying Pub. L. No. 110-329, Div. D, 122 Stat. 3574(2008), the 20 
■-'-lent of Homeland Security Appropriations Act, 2009). 
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Background (cont'd) 



Figure 1 : Airports That Are Currently Participating in or Have Received Approval from TSA to 
Participate in the SPP 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 



While TSA's study comparing the cost of screening services at SPP and non-SPP airports had some 
design strengths, such as recognizing that cost savings would be limited given the mandated 
structure of the SPP and using 2007 invoice data to determine the costs for the SPP airports, its 
usefulness could be affected by the following methodological limitations: 



1 . TSA's study design did not consider the impact of overlapping administrative personnel 
on the costs of SPP airports. TSA is currently conducting a workforce analysis to 
identify unnecessary redundancies in its staffing, including overlapping administrative 
personnel, in response to a Department of Homeland Security (DHS), Office of 
Inspector General (OIG) recommendation. According to TSA officials, this analysis is 
an ongoing effort and has no completion date. 

2. TSA underestimated costs to the government for non-SPP airports because the agency 
did not include all of the costs associated with passenger and baggage screening 
services at non-SPP airports, such as workers compensation and general liability 
insurance, and did not reflect the income received by the government from corporate 
income taxes paid by SPP contractors. Furthermore, the estimate for non-SPP airports 
does not take into account the costs of certain retirement benefits to be paid by OPM to 
TSA retirees. These costs elements can impact the reliability of the cost estimate. 
According to Statement of Federal Financial Accounting Standards No. 4, Managerial 
Cost Accounting Concepts and Standards for the Federal Government, the full cost of 
programs is be reported to assist Congress and executives in making informed 
decisions on program resources and to ensure that programs get expected and 
efficient results. 1 



'Statement of Federal Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards for the Federal Government establishes standards for 
managerial cost accounting information at federal agencies. 22 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 

TSA acknowledged that its study was narrow in scope and stated that its purpose was to 
determine the impact of the SPP on TSA's budget, rather than to determine the full cost of 
the program to the federal government or to assist Congress or executives in decision 
making. However, TSA stated that it had prepared several different versions of its cost 
comparison of SPP and non-SPP airports that were excluded from its final study report. At 
our request, TSA provided additional calculations to include some of the costs that would 
be incurred if SPP airports were federalized, as well as the income that private screening 
contractors pay in corporate taxes to the federal government when those same airports are 
operated by the contractors. These calculations showed that the cost gap between the 
SPP and non-SPP airports decreased from approximately 17.4 to 14.5 percent when the 
agency included some of these costs, such as workers compensation and general liability 
insurance, and offset tax revenues. If TSA had included the full cost of federal retirement 
in its cost comparison, the gap would decrease further. 

3. TSA's study design based SPP costs primarily on actual invoice data and non-SPP costs 
on estimates of TSA's cost in the first year of transition if the SPP airports became federal 
airports in 2007. Thus, there was more uncertainty embedded in the cost estimates for 
non-SPP airports than those for SPP airports. 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 



4. TSA's design did not include an analysis to determine how changes in underlying 

assumptions would affect the size of the estimate, and its cost comparison did not account 
for differences in screening performance. Specifically, the TSA comparison did not analyze 
the amount of costs associated with a particular level of performance. According to OMB 
guidance, analyzing uncertainty is consistent with best practices for preparing benefit-cost 
analysis of government programs or projects. 1 TSA stated that analyzing the amount of 
costs associated with a particular level of performance would be difficult and was not the 
purpose of the study. However, if quantifiable performance measures are in place, TSA 
should be in a position to compare the costs per performance measure across different 
airports to determine how the costs per each performance measure vary across airports. 
For example, performance measures, such as wait times and threat image projection 
detection rates, are quantifiable. 



1 OMB Circular A-94 (Revised) provides guidelines and discount rates for benefit cost analysis of federal programs. Among other things, OMB Circular A-76 instructs 
agencies to base decisions on competitions among public and private-sector entities in accordance with the Federal Acquisition Regulation, which allows for considering the r\A 
relative importance of both price and technical factors, commonly known as the "best value" trade-off process. The use of best value permits trade-offs between price and 
nonprice factors, allowing acceptance of other than the lowest-priced proposal. 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 



5. TSA's comparison of SPP and non-SPP airports' costs was based on fiscal year 2007 
data. However, it is unclear how representative the costs for 1 year may be. For example, 
in future years, TSA would likely incur workers compensation costs that TSA assumed 
would not be incurred the first year it took over responsibility for operating the SPP airports. 
According to GAO criteria for estimating and managing program costs, 1 estimates are time 
phased because program costs usually span many years. Depending on the activities in 
the schedule, some years may have more costs than others. Furthermore, anomalies can 
be more easily discovered when the estimate is time phased. The Catapult study 
evaluated cost data for multiple years. 2 However, in conducting its own study, the officials 
stated that TSA analyzed fiscal year 2007 data because they were the most complete 
information available for the SPP airports and that data from previous years were not 
consistent with current TSA policy or business practices. For example, according to the 
officials, the TSO workforce was structured differently in prior years, and a recent 
expansion in its part-time workforce has affected its operational costs. 



1 GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, GAO-07-1 1 34SP (Washington, D.C.: July 2007). 
2 The study showed that the cost of operating screening was generally higher at the six SPP airports for each of the years that it studied. 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 

6. TSA's study design did not call for conducting procedures needed to ensure that cost 
data collected were reliable and did not prepare documentation of its costing 
methodology called for in federal accounting standards. 1 Specific procedures and 
information used in the cost study were not described in sufficient detail to allow a 
knowledgeable person to carry out the procedures and to replicate the results in their 
entirety. 

7. Key assumptions and methods used were also not documented in sufficient detail to 
justify the reasonableness of costs. According to GAO's Standards for Internal 
Control in the Federal Government, significant events, which can include key 
decisions about assumptions and methods underlying the assignment of costs, are to 
be clearly documented. 2 TSA officials stated that the assumptions in their study were 
reviewed by multiple parties within the agency and that they could verbally "walk 
someone through" the details of their analysis. However, without sufficient 
documentation, it will be difficult for TSA to justify its decisions about assumptions and 
methods used in its costing methodology. 



'Statement of Federal Financial Accounting Standards No.4, Managerial Cost Accounting Concepts and Standards for the Federal Government. 
2 GAO-AIMD-00-21.3-1. 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 

TSA's comparison of screening performance at SPP and non-SPP airports had some strengths, 
such as excluding performance data relating to customer service because it had no assurance that 
the data were collected in a uniform manner, but its usefulness could be affected by several 
methodological limitations. 

1 . TSA's study design did not document the rationale for including the five performance 
measures reviewed, while excluding other performance measures. TSA officials stated 
that, for analysis purposes, they included the five measures that they believed were the 
most meaningful and for which they had complete data. While this is understandable, 
TSA's study provided no evidence showing that the five performance measures are the 
most meaningful indicators of performance or that they provide a comprehensive basis 
for comparing the performance of SPP and non-SPP airports. 

2. In comparing the performance of SPP and non-SPP airports, TSA's design did not 
control or otherwise account for other possible factors, such as airport configuration or 
size, that could contribute to performance differences but may not be associated with the 
SPP. However, we recognize that controlling for such factors may be difficult. 
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Methodological Limitations Could Affect the Usefulness 
of TSA's Comparison of SPP and Non-SPP Airports 
(cont'd) 

3. TSA's study design did not provide any statistical analyses to indicate the level of 
confidence in the observed differences in screening performance, consistent with 
generally accepted statistical practices. The study also did not address the procedures 
needed to ensure that performance data collected were reliable, and that there was an 
appropriate, standardized, and generally accepted methodology used for the analysis of 
the data. TSA officials stated that they did not believe that statistical analyses were 
necessary because they included only five performance measures in the study. However, 
according to OMB guidance, federal agencies should conduct a statistical and 
methodological review of all information products. Among other things, they should review 
assumptions and limitations for accuracy and appropriateness; base statements of 
comparisons and other statistical conclusions on acceptable statistical practice, and when 
performing comparison tests, test and report only the differences that are substantively 
meaningful; and ensure that data sources and technical documentation, including data 
limitations, are included or referenced. 1 



Without addressing the methodological limitations of TSA's study comparing the cost and 
performance of SPP and non-SPP airports, we could not determine the reliability of the study results. 
Reliable information on both the costs and performance of federal programs, such as the SPP, is 
essential for the effective management of government operations and decision making. Therefore, 
the TSA study should not be the sole support for major policy decisions. 



1 OMB, Standards and Guidelines for Statistical Surveys. 
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Efforts Are Under Way to Identify and Eliminate Any 
Unnecessary Redundancies at SPP Airports 



• TSA has initiated actions to identify and eliminate any unnecessary redundancies at SPP airports 
between TSA administrative staff and contractor personnel; however, these efforts are not yet 
complete. 



• In September 2006, the DHS/OIG recommended that TSA conduct a workforce analysis of 
administrative staff and develop a staffing model to identify the number of employees actually 
needed at airports. 1 



• In December 2007, the TSA contractor recommended that TSA explore reducing the redundant 
general and administrative and overhead costs at the SPP airports (see app. IV for a complete 
listing of the recommendations that the TSA contractor made regarding the SPP). 



• As an example of the importance of overhead costs and how they are allocated, at one of 
the SPP airports we visited, general and administrative and overhead costs accounted for 
half of the difference in TSA's estimated $14 million cost difference between SPP and non- 
SPP operations. 



• According to identified effective practices, strategic workforce planning requires, among other 
things, that agencies develop strategies that are tailored to address gaps in the number, 
deployment, and alignment of human capital approaches for enabling and sustaining the 
contributions of all critical skills and competencies. 2 



'Department of Homeland Security, Office of Inspector General, Review of TSA Non-Screener Administrative Positions, OIG-06-65, September 2006. 
2 GAO-04-39. 
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Efforts Are Under Way to Identify and Eliminate Any 
Unnecessary Redundancies at SPP Airports (cont'd) 



• SPP contractors and TSA have different views regarding whether unnecessary redundancies exist 
at SPP airports. 



• Private screening contractors at four of the six SPP airports we visited stated that they believed 
that unnecessary redundancies exist between supervisory TSA and contract screening personnel. 
For example, contractor officials at one airport said that TSA's screening managers perform the 
same responsibilities as the contractor's supervisors. No TSA personnel were assigned at the 
remaining two SPP airports that we visited. 



• TSA officials at three of the SPP airports we visited stated that they believed that there were no 
unnecessary redundancies in the supervisory area between TSA and contract screening 
personnel. However, a TSA official at another airport we visited suggested that duplication 
between TSA and the contractor's staff could be reduced and resources could be used more 
efficiently if the SPP concept were implemented on a regional basis. Under the present system, a 
hub airport may participate in the SPP although none of its spoke airports do so. Therefore, TSA 
may use hub airport staff to perform various tasks, such as training, for federal screening 
personnel working at the spoke airports, while the private screening contractor may hire staff to 
perform similar functions for private screening personnel working at the hub airport. 



• Our review of the organization charts and roles and responsibilities of TSA and screening 

personnel at the selected SPP and non-SPP airports revealed various staffing arrangements, but 
did not identify any obvious unnecessary redundancies. 
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Efforts Are Under Way to Identify and Eliminate Any 
Unnecessary Redundancies at SPP Airports (cont'd) 



TSA officials stated that the agency is in the process of conducting a workforce analysis to 
better understand the unnecessary redundancy concerns regarding the duties of 
administrative staff, including supervisory positions, at SPP and non-SPP airports, and develop 
a staffing model to more effectively utilize administrative staff at its airports. From November 
2006 through January 2007, TSA visited a sample of 37 airports and held focus groups with 
an additional 27 FSDs and their staff to collect data and has conducted an analysis associated 
with core functions, staffing levels, hub-spoke responsibilities, skill sets, and collateral duties. 1 
TSA analyzed the data from January 2007 through March 2007. Steps that have not yet been 
completed include providing FSDs with greater flexibility in staffing and implementing 
organizational models to ensure that there is consistency in reporting relationships and 
utilization of positions. TSA officials said that these efforts will be ongoing and collateral with 
its effort to finalize and budget for the new staffing model. 

In developing the workforce analysis and staffing model recommended by DHS/OIG, TSA is 
taking important steps to ensure that its resources are being allocated appropriately at both 
SPP and non-SPP airports. It is important that TSA continue these efforts and use the results 
of the analysis to eliminate any unnecessary redundancies and ensure that TSA staff are 
used efficiently. The issue of possible redundancies has potentially important implications for 
costs comparisons of SPP and non-SPP airports. If there are administrative staffing 
redundancies at individual airports and at TSA headquarters, the cost of the SPP program 
would be higher than necessary. 



'This sample represented 52 percent of administrative personnel nationwide; included hub, spoke, stand-alone, and privatized airports; and was based on characteristics 

that were most likely to impact staffing needs, such as airport category, hub-spoke composition, privatized versus federalized, and geographic area. 31 
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Various Factors Contribute to Airports' Decisions 
regarding SPP Participation 

• Various factors have contributed to airports' decisions about whether to participate in the SPP. 

• Managers at two of the SPP airports we visited stated that TSA had strongly suggested that 
they apply to the SPP because of constraints on the number of TSOs that TSA had at that 
time. Another airport manager said that he applied to participate in the SPP so that he could 
gain screeners at another airport, which he also managed. He was reportedly told by TSA 
that the other airport would not be staffed with additional TSA screeners due to TSA 
resource constraints, so he applied to participate in the SPP at both airports. 

• Another airport manager stated that he decided to participate in the SPP because he 
believed that private screeners would provide better customer service. 

• Managers at the remaining two airports we visited stated that they had decided to continue 
using private screeners because of their satisfactory experience with participating in TSA's 
2-year pilot program that preceded the SPP. 

• Officials at the six airports that we contacted that decided not to join the SPP stated that they 
were pleased with the federal screener workforce. In addition, officials at three of these 
airports said that they had no interest in the SPP. 
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An Association Survey Provided Views on SPP 
Participation 



• In March 2007, the Airports Council International-North America (ACI-NA) reported on the results 
of a survey it conducted to determine how to make the SPP more attractive to airports. 1 The 
survey was sent to ACI-NA airport official representatives and Public Safety and Security 
Committee airport members. ACI-NA received a total of 31 survey responses. 2 Twenty-two, or 71 
percent, of the respondents said they were not interested in participating in the SPP. The most 
frequent reasons given for this lack of interest included 

• the airport was satisfied with TSA screening services, 

• screening is a federal government responsibility, and 

• the SPP does not allow airports to have managerial control. 

• The survey respondents reported that liability and contract control and oversight were also 
important issues with respect to SPP participation. 

• The most frequently cited advantages of the SPP were staffing flexibility and customer service. 



Airports Council International-North America, Screening Partnership Program: Interest and Considerations, March 1 2, 2007. 

Due to the low response rate to the survey, the views reported in the survey reflect the views of the 31 who responded to the survey and not necessarily the views of the 
' ership as a whole. 
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Officials at Selected Aviation Associations and 
Organizations Provided Views about SPP Participation 



• Officials at all three of the aviation associations we interviewed stated that airports are concerned 
about liability associated with participating in the SPP. They said that the SAFETY Act, 1 which 
offers liability and other protection to sellers of qualified antiterrorism technologies, including 
private screening contractors, has not yet been tested and that some airport managers may 
worry that the public will hold them liable if a security incident occurs and they have chosen to 
participate in the SPP. 

• However, 49 U.S.C. § 44920(g) provides that, notwithstanding any other provision of law, an 
operator of an airport shall not be liable for any claims for damages relating to an airport 
operator's decision to submit or not submit an application to opt out of federal screening, or any 
acts of negligence, gross negligence, or intentional wrongdoing by a qualified private screening 
company or any of its employees while performing its duties under contract with TSA or by 
employees of the federal government providing passenger and property security screening 
services at an airport. 

• The aviation associations suggested that airports would be more interested in the SPP if they 
could 

• select the private screening contractor that would be awarded the passenger and 
baggage screening contract at their respective airports and 

• share in any cost savings resulting from their participation in the program. 



'Support Anti-terrorism by Fostering Effective Technologies Act of 2002, Pub. L. No. 107-296, §§ 861-65, 1 16 Stat. 2135, 2238-42. See 71 Fed. Reg. 33,147 (June 8, 2006) 
(implementing SAFETY Act regulations codified at 6 C.F.R. pt. 25). 
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Conclusions 



With regard to the SPP, as TSA acknowledged, TSA and its private screening contractors are 
constrained by both law and TSA policy, which limits opportunities to make programmatic 
changes to improve contractor performance and achieve cost savings. Therefore, it is 
understandable that differences in performance and costs of SPP and non-SPP airports may not 
vary considerably. 

While it is commendable that TSA took the initiative to conduct a study comparing the cost and 
performance of SPP and non-SPP airports, the study contained several methodological limitations 
that we identified, such as the exclusion of certain cost elements and the rationale for why certain 
performance measures were included in the study. These limitations could affect the accuracy 
and reliability of the results, and its usefulness in informing future management decisions 
regarding the SPP. In addition since TSA's workforce analysis efforts are still ongoing, it is 
unclear to what extent a reduction of any potential unnecessary redundancies could lead to 
additional costs savings in the SPP. 

As additional airports apply to participate in the SPP, it is important that TSA have accurate and 
reliable information about screening cost and performance at both SPP and non-SPP airports so 
that the agency can make informed decisions when considering whether to approve airport 
participation in the SPP. It is also important that TSA have a well-developed and well- 
documented approach for analyzing the cost and performance of SPP airports if such 
comparisons are made in the future. Because of the limitations that we have identified, the TSA 
study should not be used as the sole support for making major policy decisions. 
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Recommendation for Executive Action 



If TSA plans to rely on its study's comparisons of cost and performance of SPP and non-SPP 
airports for making future management decisions regarding the SPP, we recommend that the 
Assistant Secretary, TSA, update the study to address the methodological limitations that we 
have identified. 
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Agency Comments 



We provided a draft copy of this briefing to TSA for its review. TSA, in its written comments, 
generally concurred with our findings and recommendation and stated that its cost and 
performance analysis of private-sector airport screening will be updated as data become 
available. TSA also provided technical comments, which we have incorporated into the briefing 
as appropriate. The full text of TSA's comments is reprinted in enclosure II of our report. 
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GAO Contacts 



Should you or your staff have any questions on the matters discussed in this briefing, please 
contact Cathleen A. Berrick at (202) 512-3404 or berrickc@gao.gov or Glenn Davis at (202) 512- 
4301 or davisg@gao.gov . 
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Appendixes 



• Appendix I: TSA's Approach and Methodology for Estimating Costs of SPP and Non-SPP Airports 

• Appendix II: Methodology and Findings for the TSA Contractor's Cost Analysis 

• Appendix III: Methodology and Findings for the TSA Contractor's Performance Analysis 

• Appendix IV: Recommendations Provided by the TSA Contractor to Improve the SPP 
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TSA's Approach and Methodology for Estimating 
Costs of SPP and Non-SPP Airports 







Data source 


■ Invoice data from fiscal year 2007 for six SPP airports 


• Cost estimates were based on what TSA would have incurred if the six 
airports were run as fully federal or non-SPP airports in fiscal year 2007 


Direct labor 


Screener productive labor hours 

• Wage rates from invoice data and cost proposals 

• Includes on-site labor hours for management 

• Overtime pay 


Screener productive and nonproductive labor hours 

• Same as SPP wage data 

• Includes increased FSD staff that would be needed if SPP was not 
Dresent 

•Premium pay 


Fringe 


1 Includes FICA, 401 K, health, dental, and life insurance, 
workers compensation, flex spending account, paid holidays, 
personal time office, military leave, iury duty pay, and bonuses 


• Includes FICA, Medicare, thrift savings plan, health and life insurance, 
retirement, and awards 


3eneral and administrative 
and overhead 


■ Overhead: management support staff 

■ General and administrative: home office expenses, including 
:orporate management, accounting, human resources, 
nformation technology, and contract administration 


= ederal Security Staff Non-Personnel Compensation and Benefits, Non- 
3 ersonnel Compensation and Benefits, entire Headquarters Administration 
3 rogram, Project and Activity in the Support Appropriation, hazardous 
materials, Representation Fund, Model Workplace Non-Personnel 
Compensation and Benefits, Tort Claims, Safety Hazard Mitigation, TSO 
Scheduling Tool, and the Department of Homeland Security Working 
Capital Fund 


Other direct costs 


1 Includes consumables, recruiting, assessments, credentials, 
facilities, travel, uniforms, general liability insurance, and office 
Bquipment costs 


• Includes consumables, recruiting assessments, credentialing, facilities, 
travel, uniforms, and airport parking 


Fees 


' Includes contract fees, such as award fee, base fee, and fixed 
: ee 


• Fees not incurred for non-SPP airports 



Source: TSA. 
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Methodology and Findings for the TSA Contractor's 
Cost Analysis 

TSA's contractor used two separate models to develop its cost analysis of SPP and non-SPP airports. 

• Under the first model, the contractor compared the costs of six SPP airports with the costs of 
six comparable non-SPP airports with normalizing 1 adjustments. 

Figure 2: Airports Included in the Contractor's Cost Analysis 



SPP airport Comparable airport 

San Francisco International Airport., ► Logan International Airport 

Kansas City International Airport < ► Salt Lake City International Airport 

Greater Rochester International Airport 4 ► Syracuse-Hancock International Airport 

Joe Foss Field < ► Missoula International Airport 

Jackson Hole Airport ► Central Illinois Regional Airport 

Tupelo Regional Airport ^ Barkley Regional Airport 



Source: Catapult Consultants. 



'The purpose of data normalization (or cleansing) is to make a data set consistent with and comparable to other data used in an estimate. 
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Methodology and Findings for the TSA Contractor's 
Cost Analysis (cont'd) 



• Under the second model, the contractor used data from approximately 450 airports to compare 
the costs for each of the 6 SPP airports to a regression-based model 1 for a fully federal operation 
at the same SPP airport. That is, the model was used to determine what the SPP airport would 
cost if it used federal screeners. 



• The contractor concluded that SPP airports have historically cost more to operate than airports 
with federal screeners. According to the regression-based model, the average SPP cost 
premium is 9 percent, while the adjusted comparable model showed an average SPP cost 
premium of 17 percent. 



1 A regress ion- based model is used to predict one variable from one or more other variables. Regression models provide the scientist with a powerful tool, allowing 
predictions about past, present, or future events to be made with information about past or present events. 
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Methodology and Findings for the TSA Contractor's 
Performance Analysis 

• The TSA contractor analyzed four performance measures: 1 

1. Threat image projection detection rates 

2. Recertification pass rate 

3. Wait times 

4. Customer satisfaction 

• The TSA contractor: 

• Compared the performance of six SPP airports to six "comparable" non-SPP airports 
and to the national average for the appropriate airport category. 

• Determined that an airport was a better or worse performer by comparing SPP 
performance to its airport category average. 



Examined 4 years of performance data (fiscal years 2004 through 2007). 

ictor's analysis found that SPP a 
than those delivered 6y non-SPP airports. 



• The contractor's analysis found that SPP airports' overall performance results are equal to or better 



'According to the contractor's study, these four performance measures were nationally standardized and had large enough sample sizes to conduct meaningful analysis. 
While some of these measures were similar to those included in TSA's study, only one identical performance measure was included in both analyses. 
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Recommendations Provided by the TSA Contractor to 
Improve the SPP 



The contractor's December 2007 report, which summarized the results of its SPP analysis, 
included the following recommendations to TSA to improve the operational effectiveness and cost 
efficiency of the SPP 

1 . Explore reducing the redundant general and administrative and overhead costs at SPP 
airports. 

2. Explore the use of the SPP model as a tool to improve performance at low-performing 
fully federal airports. 

3. Explore the use of the SPP model at "hard-to-hire" airports and for airports with 
significant seasonal requirements. 

4. Explore providing additional "degrees of freedom" to SPP contractors to foster 
innovation, superior performance, and cost controls. 
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Recommendations Provided by the TSA Contractor to 
Improve the SPP (cont'd) 



TSA officials stated that they have not committed to implementing the recommendations made 
by the contractor. 

TSA officials informed us that the agency maintains a neutral position on the SPP and neither 
encourages or discourages airports from applying to participate in the program. 
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Related GAO Products 



• Aviation Security: TSA's Staffing Allocation Model Is Useful for Allocating Staff among Airports, 
but Its Assumptions Should Be Systematically Reassessed. GAO-07-299. Washington, D.C.: 
February 28, 2007. 

• Aviation Security: Progress Made to Set Up Program Using Private-Sector Airport Screeners, but 
More Work Remains. GAO-06-166. Washington, D.C.: March 31, 2006. 

• Aviation Security: Preliminary Observations on TSA's Progress to Allow Airports to Use Private 
Passenger and Baggage Screening Services. GAO-05-126. Washington, D.C.: November 19, 
2004. 
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Enclosure II: Comments from the 
Department of Homeland Security 



U.S. IJspartmenf of l'fomeEand -SfeurftJ' 
Washington. DC 20528 




Homeland 
Security 



December 30, 2008 



Ms. Cathleen A, Berrick 

Director, Homeland Security and Justice Issues 

U.S. Government Accountability Office 

441 G Street, NW 

Washington, DC 20548 

Dear Ms. Berrick: 

RE; Draft Report GAO-09-27R, Aviation Security. TSA's Cost and Performance 
Study of Private-Sector Airport Screening (GAO Job Code 440647) 

The Department of Homeland Security appreciates the opportunity to review and 
comment on the draft report referenced above. Die report documents the Government 
Accountability Office's (GAO) analysis of the Transportation Security Administration's 
(TSA) cost and performance study of Screening Partnership Program (SPP) airports, 
actions TSA has taken to identify and reduce personnel redundancies, and factors that 
contribute to an airport operator's decision about whether to participate in the SPP, 

TSA appreciates that GAO recognized the underlying constraints when analyzing cost 
and performance of SPP and non-SPP airports. GAO did not identify any redundancies 
between administrative staff assigned to SPP and non-SPP airports, and noted TSA's 
efforts to identify any such redundancies. Finally, GAO found that various factors 
contribute to airports' decisions to participate in SPP. 

GAO noted some limitations of the cost and performance study of SPP due to study 
purpose and data availability, and recommends that the study be revised to account for 
those limitations if TSA plans to use the study for SPP decision making. I SA officials 
concur with this recommendation, and as the report notes, update the analysis as data 
becomes available. 



Sincerely, 




Director 

Departmental GAO/OIG Liaison Office 



VrTVW.dhs.gllV 



(440647) 
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